fix(watchdog): skip cron-liveness rescue when daily-regen is disabled - #10540
Conversation
The liveness check in section C treats a >10 h gap in daily-regen runs as a starved GitHub schedule and re-dispatches the workflow. A manually disabled daily-regen produces exactly that signature — its cron never ticks, so the gap grows without bound — but `gh workflow run` on a disabled workflow returns HTTP 422. Under `set -euo pipefail` that killed the whole scan step after sections A and B had already done their work, so every scan outside the quiet window reported failure (4 of the last 6 runs). Read the workflow's state before the rescue and skip with a notice unless it is `active`. A deliberate pause stays paused; genuine scheduler starvation on an active workflow is still rescued. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, matches the PR’s stated failure mode, and avoids failing the watchdog scan when daily-regen is intentionally disabled.
Pull request overview
This PR prevents the scheduled watchdog workflow from failing when it attempts to “rescue” the daily-regen cron while that workflow is manually disabled, by checking the target workflow’s state before dispatching.
Changes:
- Add a pre-check in the watchdog cron-liveness section to read
daily-regen’s Actions workflowstateand skip the rescue when it isn’tactive. - Add a
[Unreleased]changelog entry describing the watchdog fix and the prior failure mode (HTTP 422 on dispatching a disabled workflow).
File summaries
| File | Description |
|---|---|
| CHANGELOG.md | Documents the watchdog fix under [Unreleased] → Fixed. |
| .github/workflows/watchdog-stuck-jobs.yml | Skips cron-liveness rescue dispatch when daily-regen is not active, avoiding false watchdog failures. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, correctly avoids the known HTTP 422 failure mode for disabled workflows, and preserves the existing rescue behavior for active schedules.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, correctly avoids dispatching a disabled workflow (preventing the observed HTTP 422 failure under set -e), and preserves the existing rescue behavior for active workflows.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, correctly guards the gh workflow run path when daily-regen.yml is not dispatchable, and includes a corresponding changelog entry.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The change is small, well-scoped to the failing rescue path, and preserves the existing behavior for active workflows while preventing false-negative watchdog failures.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, matches the PR description, and prevents a confirmed failure mode without altering the existing rescue behavior for active workflows.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, matches the stated failure mode, and safely avoids a known gh workflow run failure path without altering the successful watchdog scans (sections A/B).
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, well-scoped to the failing watchdog path, and safely avoids a known gh 422 failure mode while preserving the existing rescue logic for active workflows.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
…abled-daily-regen # Conflicts: # CHANGELOG.md
#10628) ## Why The `babysit-pipeline` skill covers watching one fresh spec to 15/15. Backfilling coverage gaps across the catalogue — dozens of older specs each missing the libraries added after they were generated — is a different job, and the skill said nothing about it. Running it by hand on 2026-08-24 surfaced four traps that cost, or nearly cost, real implementations. ## What's new **Section 5 · Gap backfill** - Compute the missing set from `plots/{spec}/metadata/{lang}/{lib}.yaml` on `origin/main`, **not** from `impl:{lib}:done` labels — most affected specs have closed issues, so their labels are absent or stale. Includes the `git ls-tree | awk` one-liner. - `run_spec.sh <spec> <model> <lib>...` as the driver: staggered dispatch, poll to metadata, `RESULT=COMPLETE|PARTIAL|TIMEOUT`, skips libraries already on main so a re-run retries exactly the gaps. - Two specs in parallel (~4 specs/h vs ~2; ten concurrent `impl-generate` runs showed no rate-limit effects). - A `done.log` / `deferred.log` ledger written *before* dispatching the next spec, so a compaction or crashed session resumes without recounting. - **The one-retry rule**, with the evidence for why it is not optional: the single retry recovered `highcharts/treemap-basic`, `ggplot2/wireframe-3d-basic` and `ggplot2/network-force-directed`, all three of which had already been read as capability gaps. **Four new gotchas** | Trap | Why it matters | |---|---| | `Marking <lib> as failed: N generation attempts` counts more than this run | The markers span past campaigns (#10627 scopes this to 12 h). Check the `Previous failures for <lib>/<spec>: N` notice before concluding a library can't do a plot type. | | `impl:<lib>:failed` is terminal **and** unreliable | Nothing re-dispatches it — watchdog case 3 fires once, then only logs `needs manual attention`. And of 87 such labels, **42 sat on implementations that had since landed**. | | "Agent reports success, writes no file" | 8 of 85 generate runs (~9%). Self-heals when retry budget remains (`bubble-basic/highcharts`: failed 17:55, succeeded on auto-retry 18:02). One occurrence is noise. | | Static library + interactive/3D spec | The one gap shape that usually is genuine — 18 of the 45 real gaps. Still give the one retry, then defer. | **`run_spec.sh` joins the bundled scripts.** Its hardcoded `REPO=/home/tirao/anyplot` is replaced by resolution from the script's own location (`ANYPLOT_REPO` overrides), matching how `poll_spec.sh` derives `HERE`. ## Verification - `bash -n` on `run_spec.sh`; repo auto-resolution checked from the skill directory (`git -C "$HERE" rev-parse --show-toplevel` → the repo root). - The skill's existing `ALL_LIBS` gotcha now names `run_spec.sh`'s `lang_of` too, so a library addition updates both scripts. - Content is drawn from a real backfill run, not invented: every number above has a corresponding workflow run or label query behind it. ## Related - #10627 fixes the retry-cap bug the first gotcha describes. - #10540 fixes the watchdog crash found in the same session. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
The change is minimal, matches the stated failure mode, and safely avoids failing the watchdog scan when the target workflow cannot be dispatched.
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0 new
- Review effort level: Lite
…er's fetch (#10660) ## Why #10628 documented gap backfill while the backfill was still running. Finishing it falsified three of its claims, so this PR corrects them rather than leaving confident wrong guidance in a skill. Running the backfill in two parallel slots also exposed a race in the driver, fixed here. **Scope: prose plus one executable change.** `SKILL.md` and `CHANGELOG.md` are documentation; `run_spec.sh` gains a fetch retry (behaviour change, details below). ## Corrections **1. Halt-on-cluster counted the wrong thing.** The threshold was "≥5 failed `Generate:` runs in minutes = quota exhausted". Since #10627 restored the full retry budget, a single impossible pair spends three runs on its own auto-retries — so two bad pairs trip a raw count of five while the pipeline is perfectly healthy. That happened during the run: 6 failures at 22:29–22:37, all of them `plotnine` on two specs, nothing wrong with the pipeline. Now counts distinct `(spec, library)` pairs. **2. "Static library + interactive/3D spec is the one gap that is usually real" was backwards.** Every category-level prediction made during the backfill turned out wrong: | Prediction | Outcome | |---|---| | chartjs can't do treemap / sankey | chartjs succeeded on `network-force-directed`, `arc-basic`, and every JS-block spec | | plotnine can't do 3D | `bar-3d-categorical` succeeded; `scatter-3d` did not | | ggplot2 can't do wireframe | succeeded on retry | | pygal is a lost cause (6 failed labels) | succeeded on `map-marker-clustered` and `line-stress-strain` | 17 of 20 parked pairs generated fine. Exactly three failed under a full budget — plotnine on `scatter-3d`, `contour-3d`, `line-3d-trajectory`, all of which need a spatial projection plotnine does not have, while the "3D" spec representable in 2D went through. The gotcha now says: measure, don't predict. **3. The one-retry rule needed a precondition.** The workflow now spends three attempts per campaign by itself, so a pair that comes back missing may already be measured and the manual retry adds nothing. The skill now shows the actual command — `gh run list` has no per-spec filter, so it filters the output by run title with jq. Three failures minutes apart is a gap, one is a flake. The same check resolves `RESULT=TIMEOUT` with `recent generate failures: 0`, which only means the failures aged out of the driver's 25-minute window (seen on `line-3d-trajectory`, which had in fact failed three times 40 minutes earlier). ## New gotcha Spec IDs harvested from `impl:*:failed` issue titles must be intersected with the real `plots/` directories: **14 of 26** pointed at specs that no longer exist on main, and such a dispatch dies seconds in at `Validate specification exists`. This is how the rescue list first read as 45 missing implementations when only 20 were real. ## Behaviour change: `run_spec.sh` retries its fetch Two drivers polling the same checkout collide on the ref lock: ``` error: cannot lock ref 'refs/remotes/origin/main': is at 6b666c1 but expected c736a84 ``` A lost fetch leaves `origin/main` stale, so `meta_present` understates what has landed and the poller reports `PARTIAL` for libraries that are already merged — a failure that disguises itself as a stalled spec. `fetch_main()` now retries three times with backoff and, when all three lose, logs the warning **with git's own stderr** so a ref-lock collision is distinguishable from an auth or network failure. ## Verification - `bash -n` on `run_spec.sh`; the failure path exercised against a bad ref → `last error: fatal: couldn't find remote ref …`, confirming the message survives to the log. - Every number in the corrections traces to a workflow run or label query from the 2026-08-24 backfill; the ledger is in `agentic/runs/babysit-2026-08-20/`. ## Related - #10627 (retry cap) — verified in production: `Previous failures for plotnine/contour-3d since 2026-08-24T10:38:22Z: 2` followed by `3 failed attempt(s) in the last 12h (cap: 3 per campaign)`. - #10628 — the section this corrects. - #10540 — verified by dry run: `daily-regen liveness: workflow state is 'disabled_manually', not active — rescue skipped`, scan completing successfully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Problem
Watchdog: scan (cron)has been failing on 4 of its last 6 runs (2026-08-22 → 2026-08-24). The failure is always the last thing the step does:daily-regen.ymlis currentlydisabled_manually. A disabled workflow has exactly the signature the section C liveness check looks for — its cron never ticks, so the gap grows without bound — but dispatching it is impossible, and the resulting non-zeroghexit kills the step underset -euo pipefail. Sections A and B had already completed their scans by then, so the actual watchdog work happened and was then reported as a failed run.Fix
Read the workflow's state before attempting the rescue and skip with a notice unless it is
active:A deliberate pause stays paused; genuine scheduler starvation on an active workflow is still rescued exactly as before. The
|| echo unknownfallback means an API hiccup also skips the rescue rather than failing the scan.Verification
yaml.safe_loadparses the workflow.run:block extracted and checked withbash -n.if→elif.Note for the reviewer
Whether
daily-regenshould be re-enabled is a separate, human decision — this PR only stops the watchdog from crashing on the answer being "no".🤖 Generated with Claude Code
https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP